CO; 



xn 



J gederick R*Fish 
- 1855-1930 

WK. Richardson 
1859-1951 



BOSTON 
NEW YORK 
SILICON VALLEY 
SatfrHERN CALIFORNIA 
f i TWIN CITIES 

WASHINGTON, DC 



Fish & Richardson p.c. 



/ 

4225 Executive Square 
Suite 1400 
La Jolla, California 
92037 o S 



Telephone ^^" ^ 
619 678-5070 



CM 
CO 



December 9,1998 

Attorney Docket No.: 08305/050001 

Box Patent Application 

Assistant Commissioner for Patents 
Washington, DC 20231 

Presented for filing is a new original patent application of: 



Facsimile 

619 678-5099 m<y^ 
U i 

Web Site -n ' 
www.fr.com 



SON 
= 0 



sCM 



Applicant: 
Title: 



MICHAEL KAPLINSKY 

COLOR CORRECTION OF MULTIPLE COLORS USING A 
CALIBRATED TECHNIQUE 



Enclosed are the following papers, including all those required to receive a filing date 
under 37 CFR§ 1.53(b): 



Specification 

Claims 

Abstract 

Declaration 

Drawing(s) 

Enclosures: 

• Postcard. 



Pages 

14 

4 

1 

[To Be Filed At A Later Date] 
3 



This application is entitled to small entity status. A small entity statement will be filed 
at a later date. 



"EXPRESS MAIL" Mailing Label Number E Yh ^ ^2'7 <^ *a 3 3 ^lO ^ 
Date of Deposit \2,^Qf.l9^ 

I hereby certify under 37 CFR 1 . 10 that this correspondence is being 
deposited with the United States Postal Service as "Express Mail Post 
Office To Addressee" with sufficient postage on the date indicated 
above and is addressed to the Assistant Commissioner for Patents, 
Washington, D.C. 20231. 



Fish & Richardson p.c. 



BOX PATENT APPLICATION 
December 9, 1998 
Page 2 



Basic filing fee 

Total claims in excess of 20 times $9.00 
Independent claims in excess of 3 times $39.00 
Multiple dependent claims 
Total filing fee: 



$0.00 
0.00 
0.00 
0.00 

$0.00 



No filing fee is being paid at this time. Please apply any other required fees, 
EXCEPT FOR THE FILING FEE, to deposit account 06-1050, referencing the 
attorney docket number shown above. A duplicate copy of this transmittal letter is 
attached. 

If this application is found to be INCOMPLETE, or if a telephone conference would 
otherwise be helpful, please call the undersigned at 619/678-5070. 

Kindly acknowledge receipt of this application by retuming the enclosed postcard. 

Please send all correspondence to: 

Scott C. Harris 

Fish & Richardson P.C. 

4225 Executive Square, Suite 1400 

La JoUa, CA 92037 




Reg. No. 32,030 



Enclosures 



APPLICATION 



FOR 

UNITED STATES LETTERS PATENT 



TITLE: COLOR CORRECTION OF MULTIPLE COLORS USING A 

CALIBRATED TECHNIQUE 

APPLICANT: MICHAEL KAPLINSKY 



"EXPRESS MAIL" Mailing Label Number ^^V^5X?4U3Sg>|tX 3 

Date of Deposit 

I hereby certify under 37 CFR 1.10 that this correspondence is being 
deposited with the United States Postal Service as "Express Mail Post 
Office To Addressee" with sufficient postage on the date indicated 
above and is addressed to the Assistant Commissioner for Patents, 
Washington, D.C. 20231. 



PATENT 

ATTORNEY DOCKET NO: 083 05/050001/98-2 6 



COLOR CORRECTION OF MULTIPLE COLORS 
USING A CALIBRATED TECHNIQUE 

Background 

A typical color camera separately monitors red, green, and 
blue components of an image. Electronic imagers rely on some 
means of separation of the illumination incident on the sensor 
into a number of spectral channels. Widely-used means of 
discrimination between spectral components of the imaging scene 
include the Color Filter Arrays (usually employed in systems 
based on single focal plane array) and beam- splitting prisms 
(primarily employed in systems based on multiple focal plane 
arrays) . In either case, the incident illumination is separated 
into a small number, usually 3 or 4 , discrete color (spectral) 
channels . 

For any given scene, the exact ratio of color channel values 
will depend on the spectral characteristics of the entire optical 
stage of the imager. This is typically done using a color filter 
array over a photosensitive area. The color filter array is 
formed of a plurality of different colored elements, which 
respectively pass only color of a predetermined spectral 
parameter. 

A typical color filter array is shown in FIG. lA. Each of 
the boxes such as 100 represents a single pixel . Each set of 
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four boxes outlined by the line 102 can be considered as a 
megapixel. The pattern in the megapixel repeats throughout the 
entire color filter array grid. 

Spectral channels of the modern color imagers do not have 
5 the same spectral sensitivities as color-sensitive elements, the 
cones, of the human eye. FIGs. IB - ID respectively illustrate 
the spectral sensitivity curves of the human eye {^,y,p) and 
spectral transmittances of the red, green and blue color channels 
O typical for modern RGB electronic imaging systems. As a 
iffi consequence, the direct use of individual color channels values 
.yp of the imager as stimuli for typical display devices, such as VGA 
rij or NTSC monitors, does not lead to correct color rendition. 
Hence, colors are corrected color signals detected by the 
electronic imaging system into color channel stimuli appropriate 
1%;' for the output to the image rendering device. 

Each megapixel 102 has two green filters, one red filter, 
and one blue filter. This is because the eye is usually more 
sensitive to green than it is to red and blue. 

The intent of the filter of FIG. lA is to provide an image 
20 which is precisely matched to the spectral content of the eye. 

However, this filter, while it does the best that it can, is not 
precisely matched. It is often desirable to interpolate between 
the values . 
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For example, the value received at area 10 0 is only 
indicative of the green portion impinging on area 100. However, 
some part of that incoming light is also red. Another part of 
the incoming light is also blue. Hence, each of the pixels is 
5 processed according to a transformation to solve the equation: 



R = KiiR + 
G = K21R + 
B = K3-LR + 



+ K13B 
K22G + K23B 

RnnG + KotB 



This transformation completely defines the system. This 
ifll includes the so-called color transformation matrix: 







^12 










^23 












K31 


^32 


K33 



15 Summary 

The inventor recognized from the above, however, that this 
corrects only for red, green, and blue parts of the image. The 
actual red, green, and blue colors look good. The other colors 
are not corrected quite as well. 
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The present system corrects the signal for all of a 
plurality of desired colors. This is complicated, however, by 
the fact that one only has control over red, green, and blue. 

According to the system as disclosed herein, the operation 
is broken down into a number of colors, e.g., 24 colors. The 
camera is used to detect a color chart or a chromaticity chart 
which has the 24 colors of interest. Each detected color is 
compared with a reference color. The coefficients of the color 
coefficient matrix are adjusted to make each of those colors 
approach the reference color more exactly. Another aspect 
weights the colors so that the most important colors obtain more 
correction . 

Brief Description of the Drawings 
These and other aspects of the invention will be described 

in detail with respect to the accompanying drawings, wherein: 
FIG. 1 shows a prior art megapixel system used a color 

filter array; 

FIGs. IB-ID illustrate the sensitivity of the eye to 
different colors ; 

FIG. 2 shows a basic setup of the system of the present 
embodiment ; and 

FIG. 3 illustrates a typical calibration chart. 
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Description of the Preferred Embodiments 

The conventional techniques to determine the coefficients 
Kij of the color correction matrix are based on analyzing the 
image of color calibration chart with N distinct color elements. 
In order to calculate a color correction matrix from the image of 
the color calibration chart, the appropriate stimuli Yi, suitable 
for driving a given image rendering device, must be known for 
each element of the chart from an independent experiment. Such 
color measurement techniques are, for example, specified by the 
Commission on Illumination (CIE) . Each of N distinct elements of 
the calibration color chart Yi are known. Xi are determined 
based on imaging of the chart. The coefficients of the color 
correction matrix solve linear equations or solve N equations 
for each chart element . 

Color correction matrix obtained by this technique provides 
for correction of color rendition of all N colors used in the 
process, provided that the colors could be physically displayed 
by the image rendering device and that they are within the 
spectral sensitivity range of the imaging system. 

Even though the color correction matrix obtained by this 
approach provides for good rendition of the N colors used in 
calibration, other target colors are not rendered correctly. 

In fact, in many real imaging systems once the basic colors 
(such as red, green and blue in RGB systems) have been 
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calibrated, the white elements of the scene appear as pink or 
blue. To counteract this effect, many electronic imaging systems 
employ white balancing techniques subsequently to color 
correction. White balancing algorithms aim at achieving good 

5 rendition of white color by adjusting (dynamically or statically) 
the gains on individual color channels. This procedure, in 
effect, modifies the coefficients of the color correction matrix. 
However, modification of the color correction matrix inevitably 

tfJ leads to deterioration of the rendition quality of the basic 
liffi colors used in color calibration. For example, in RGB systems, 

the good quality of rendition of white is achieved at the expense 

fy of somewhat arbitrary degradation of red, green and blue. In 

y: extreme cases, white balancing may lead to the situation where 

only white color is rendered correctly and most other colors are 
i^ visibly distorted. 

^ In general, it is difficult to achieve ''perfect" color 

rendition across the entire visible spectrum using electronic 
imaging system with finite number of color channels. Since the 
spectral sensitivities of realistic imaging systems and the human 
20 eyes do differ at every wavelength, every color representable by 
the image rendering device would have to be corrected 
individually. For example, a VGA monitor with 24 bits per pixel 
(8 bits per color channel) would require color correction matrix 
with 2^^ rows to achieve the best possible color rendition. 
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This system selects a number of colors including at least 
the primary colors (3) , white, and at least 1-2 0 other colors, 
preferably at least 3 others. This generates the NxN color 
correction matrix for any given N- channel imaging system that 
would yield the best possible compromise in quality of the 
rendition of the large numbers of colors, as opposed to 
perfecting rendition of N colors at the expense of others. All 
of these colors must be balanced. 

The preferred setup is shown in FIG. 2. A color chart 200 
has thereon a plurality of desired colors. While the preferred 
embodiment uses 24 colors, any number could be alternately used. 
Any number of colors, greater than the three primary colors (red, 
green, and blue or cyan, magenta, yellow) could be used. A video 
camera 202 obtains an image of the color chart 200, which is 
processed in the image processor 204. Preferably those other 
colors include at least white, and at least three other preferred 
colors, but more preferably the other colors include 20 other 
colors total for a total of 24 colors. 

Image processor 204 compares each detected color with a 
reference color , 

The color calibration chart has a number of distinct color 
elements corresponding to the colors that are deemed important 
for overall good color rendition of the electronic imager. For 
an N- channel imager, the proposed technique does not impose any 
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limitation on the number of colors to be optimized, as long as 
the calibration table contains at least N+1 distinct color 
elements. The color channel stimuli Yi appropriate for the given 
image rendering device (such as RGB values for VGA monitor) 
should also be known for each color element of the calibration 
chart from an independent experiment . 

The techniques for characterizing the calibration chart are 
beyond the scope of the present disclosure. There is a number of 
commercially-available and well known color calibration charts 
characterized by their manufacturers in accordance with standard 
techniques set by standard-defining organizations such as Inter- 
Society Color Council at National Bureau of Standards (ISCC) and 
International Commission on Illumination (CIE) . 

In order to generate an optimal color correction matrix, the 
calibration chart is imaged under illumination conditions similar 
to those that will be used during normal operation of the imager. 
Once the image of the chart is obtained, the detected signal 
values of each color channel Xi of the imager are recorded for 
all color elements of the calibration chart. 

For each of the colors to be corrected, three least -squared 
fits are carried out. The least-squared fits are done for each 
of the primary colors; here, red, green, and blue. An error 
signal G^ is obtained by taking the square of the difference 
between Gn' (what one expects to see for the green color) 
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subtracted from the G actual (what one actually sees) . Similar 
operations are done for red and blue. 

The mathematical representation of this operation is shown 
in Equation 3 . 



(Gn' [what expect to see] - G^ [actual] ) ^ = Gg 



(Rn' - Rc)2 = Rg 



. . (3) 



(Bn' 



B. 



^ The goal of this technique is to minimize G^, Re, and for 

'Sj all of the 24 colors, all for all of the desired colors for which 

lH correction is to be carried out. The coefficients in the color 

M= correction matrix of Equation 2 are adjusted in order to minimize 

□ the 24, three-item sets. This can be done by trial and error, or 

^ by solution using linear equations to dynamically change all the 
values until the best mix is reached. The minimization can be 

15 expressed mathematically as 



mm 



where K is the number of the color channel currently being 
calibrated, M is the number of color chart elements used for 
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calibration, and N is the number of the color channels in the 
imaging system , 

Hence, this system adjusts all the values until making the 
best compromise . 

A second embodiment notes that color calibration based on 
Equation (2) results in the color correction matrix that yields 
comparable quality of color rendition for all colors in the 
calibration chart. However, the distortion of some colors 
impacts subjective image quality more than the distortion of 
other ^^less important" colors. For example, poor color rendition 
of white, gray and skin colors is more noticeable than poor 
rendition of blue and yellow parts of spectra. 

This embodiment, therefore, modifies the color calibration 
process to attach more importance to some colors at the expense 
of the others. In the context of proposed least-squares based 
method, the ability to prioritize colors is achieved by using 
weighting coefficients multiplying individual color deviations. 

Therefore, in this embodiment, the color correction matrix 
is evaluated based on minimization of the weighted sum of squares 
of differences between detected color channel values Xi and 
corresponding stimuli Yi across entire set of color elements of 
the calibration chart as follows: 
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M / 1\ T \ 



C^^sC^ 5...5 Cjj^jy-, ^ l^***N 



where Wj is the weight given to the color of the j-th element of 
the calibration color chart. 

This uses the equations shown as Equation 4. 



(Gn' - Ge)' • = G^ 

(Rn' - Rc)^ • Wi = Re ... (4) 



Where J is between 1 and 24 and each of the X' s representing a 
different color, this second embodiment adds a weighting factor 
W^. Each color is weighted according to its importance in seeing 
an image the way the eye expects to see the image. Practically 
speaking, red, green, and blue are extremely important, white is 
important, skin color and gray scale are also important. 
Unimportant ones are brown and other dull colors. All of the 
rest are medium importance. 
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Since this system takes care of all colors simultaneously, 
one obtains the best possible trade-off between all of the 
different possibilities. 

A preferred mode uses a 3 -color RGB Byer pattern color 
filter assembly. The system uses 24 -color GretagMacbeth 
ColorChecker chart marketed by GretagMacbeth, New Windsor, NY as 
the color calibration chart. GretagMacbeth describes their chart 
as "Checkerboard array of 24 scientifically prepared color 
squares in a wide range of colors . Many of these squares 
represent natural objects of special interest, such as human 
skin, foliage and blue sky. These squares are not only the same 
color as their counterparts, but also reflect light the same way 
in all parts of the visible spectrum. Because of this unique 
feature, the squares will match the colors of natural objects 
under any illumination and with any color reproduction process." 

This preferred mode assigned higher weights to chart 
elements corresponding to red, green, blue and human skin 
elements as well as a number of greyscale elements. 

FIG. 3 shows an exemplary ColorChecker chart. 

Another aspect of the present specification is the approach 
to white balance procedures in conjunction with use of the color 
correction matrix generated by described above method. As 
mentioned in the Background, the conventional approach to white 
balancing is to adjust the pre-computed color correction matrix 
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SO that the image areas corresponding to white colors appear 
white. This is usually done by adjusting individual gains on the 
color channels of the imager to achieve equality of all color 
components for white areas of the image. However, the byproduct 
of this procedure is that the quality of the color rendition for 
colors used during calibration of the color correction matrix is 
being somewhat arbitrarily compromised. In contrast, the color 
correction method disclosed here results in optimal balance 
between quality of rendition of white as well as other important 
colors that were given high weights during color calibration. 
Thus optimal white balance is "built-in" the color correction 
matrix and does not require further adjustments as long as the 
spectra of the illumination does not change. 

Another important reason for using dynamic real-time white 
balancing in imaging systems is to compensate for possible 
changes in the illumination spectra. From this point of view, 
dynamic white balancing will be beneficial to systems calibrated 
according to the present invention as well. However, since the 
initial color correction matrix determined from Equation (3) 
already provides white balance under the same illumination 
spectra as was used for calibration, the traditional methods of 
dynamic white balancing should be modified. 

In systems calibrated in accordance with the present system, 
the aim of the dynamic white balancing is to keep ratios of the 
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color channels of the imager at white areas of the image equal to 
the ratios of the same color channels measured for white image 
areas in the same illumination conditions as existed during 
calibration. This approach allows to dynamically compensate for 
changing illumination spectra without favoring the quality of 
rendition of white at the expense of other colors. 

Although only a few embodiments have been described in 
detail above, those of ordinary skill in the art will understand 
that modifications are possible without departing from the 
teaching noted above. 
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What is claimed is: 

1. A color correction method comprising: 

obtaining reference outputs from an image sensor using a 
color image array, said reference outputs being indicative of 
outputs for a plurality of known reference colors, said plurality 
including at least 3 primary colors, and at least 2 other colors 
other than said primary colors; 

determining an error measure between said outputs and what 
would be expected for each of said outputs; 

obtaining an color correction matrix that minimizes said 
error measure for each of said plurality of colors; and 

using said color correction matrix to simultaneously 
optimize the system for each of said plurality of colors in said 
color reference chart and to obtain a color-corrected image. 

2. A method as in claim 1 further comprising weighting 
certain ones of the plurality of colors to be more corrected than 
other ones of the colors. 
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3 . A method as in claim 1 wherein said comparing comprises 
obtaining, for each of the plurality of colors, 

(Gn' [what expect to see] - Gc [actiial] ) ^ = Gg 

(Rn' - Rc)^ = Re 
(Bn' - Bc)^ = Be 

and minimizing Gg, Re/ and Bg for each of the plurality of colors. 

4. A method as in claim 3 wherein there are at least 7 
colors . 

5. A method as in claim 3 wherein there are 24 colors. 

6. An image sensor apparatus, comprising: 

an image sensor device, operating using a color filter array 
which provides color filtering such that each pixel must be 
interpolated to determine all color components that actually 
impinge on an area of said pixel; 

an image interpolater , operating according to a color 
correction matrix, said color correction matrix being adjusted 
according to at least 3 primary colors, and at least two 
additional colors other than said primary color. 

7. An apparatus as in claim 6 wherein said color 
correction matrix is adjusted according to at least 3 primary 
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colors, white, and at least 3 colors other than said 3 primary 
colors and white. 

8. A device as in claim 6 wherein said adjustment is 
according to a total of 24 colors. 

9. A system as in claim 6 wherein said operation operates 
according to 

(Gn' [what expect to see] - [actual] ) ^ = Gg 

(Rn' - Rc)^ = Re 
(Bn' - Be)' = Be 

10. A system according to claim 6 wherein said color 
correction matrix has some colors weighted more than others of 
the colors. 

11. A system as in claim 10 wherein red, green, and blue 
are weighted higher than dull colors such as brown. 

12. A system as in claim 6 wherein said color correction 
matrix is adjusted according to all colors of a chromatacity 
chart . 
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13, A method of correcting an image from an image sensor, 
comprising: 

dividing the image sensor into a plurality of pixels; 

placing color separators over said plurality of pixels, such 
that each pixel receives incoming light that is filtered to 
emphasize one color component; 

obtaining a correction matrix for said pixels, said 
correction matrix being one which takes into account correction 
of incoming radiation for at least 3 primary colors, and 2 other 
colors other than said primary colors. 

14. A method as in claim 13 wherein said other colors are 
weighted so that said correction matrix corrects for some of said 
colors more than for others of said colors. 
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TECHNIQUE FOR COLOR CORRECTION 



Abstract of the Disclosure 
A system of correcting for color filter array by correcting 
for a plurality of colors in the color filter array. A color 
correction matrix is formed which corrects each color for a 
plurality of desired characteristics simultaneously. These 
desired characteristics can include, for example, all 24 colors 
in a chromaticity chart. An additional aspect includes weighting 
the more important colors relative to the less important colors. 
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